Skip to content

Replace dynamodb scan with query #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
goruha wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Replace dynamodb scan with query #31

goruha wants to merge 12 commits into from

Conversation

@goruha
Copy link
Member

@goruha goruha commented Jun 12, 2024 •
edited
Loading

what

  • Replace DynamoDB scan with query

why

  • Because scan works by batch 1Mb of data.
If the total size of scanned items exceeds the maximum dataset size limit of 1 MB, the scan completes and results are returned to the user. The LastEvaluatedKey value is also returned and the requestor can use the LastEvaluatedKey to continue the scan in a subsequent operation.
  • Reduce cost. query is cheaper then scan
  • Fix false negatives get plan cases

CleanShot 2024-06-12 at 14 33 55@2x

references

Migration

  • Create dynamodb index ``
          - name: commitSHA-index
            hash_key: commitSHA
            range_key: createdAt
            projection_type: ALL
            non_key_attributes: []
            read_capacity: null
            write_capacity: null

@goruha goruha requested review from a team as code owners June 12, 2024 12:35
@goruha goruha requested review from hans-d and kevcube June 12, 2024 12:35
goruha and others added 3 commits June 12, 2024 12:36
@goruha @github-actions
Update contents of the dist directory
e66d0ec
@goruha
Update workflows
38ed404
@goruha
Merge branch 'replace-dynamodb-scan-with-query' of github.com:cloudpo…
2c73d5b 
…sse/github-action-terraform-plan-storage into replace-dynamodb-scan-with-query

* 'replace-dynamodb-scan-with-query' of github.com:cloudposse/github-action-terraform-plan-storage:
  Update contents of the dist directory
@goruha goruha added the major Breaking changes (or first stable release) label Jun 12, 2024
osterman
osterman previously approved these changes Jun 12, 2024
View reviewed changes
@goruha goruha added the do not merge Do not merge this PR, doing so would cause problems label Jun 13, 2024
mcalhoun
mcalhoun suggested changes Sep 19, 2024
View reviewed changes
src/modules/terraformPlan/repo/DynamoDbMetadataRepo.ts Outdated
KeyConditionExpression: "#commitSHA= :commitSHA",
FilterExpression:
"#owner = :owner and #repo = :repo and #commitSHA = :commitSHA and #component = :component and #stack = :stack",
"#owner = :owner and #repo = :repo and #component = :component and #stack = :stack",
Copy link
Contributor

@mcalhoun mcalhoun Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we removing the condition to match the commit SHA? I don't see how this could be correct?

Copy link
Member Author

@goruha goruha Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mcalhoun because we filter sha with KeyCondition
https://github.com/cloudposse/github-action-terraform-plan-storage/pull/31/files#diff-95366d8215594fbafa600c4d01acd3040d43de241933dcf7a6ad86ead80e391aR33

KeyConditionExpression: "#commitSHA= :commitSHA",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@osterman osterman osterman left review comments

@hans-d hans-d Awaiting requested review from hans-d hans-d is a code owner automatically assigned from cloudposse/contributors

@kevcube kevcube Awaiting requested review from kevcube kevcube is a code owner automatically assigned from cloudposse/contributors

+1 more reviewer

@mcalhoun mcalhoun mcalhoun requested changes

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

do not merge Do not merge this PR, doing so would cause problems major Breaking changes (or first stable release)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@goruha @osterman @mcalhoun